1. Field of the Invention
This invention relates to performing diagnostics on computer systems. More specifically, this application relates to remotely testing computer systems having a variety of hardware/software combinations.
2. Background of Related Art
Traditional testing of computer systems and their associated software and firmware typically require that manual tests be designed in order to make sure that each function on given programs is operating as expected. In addition, such testing must be performed on a variety of systems running different hardware/software combinations to determine the compatibility of the underlying system. These tests require substantial user intervention because certain operations on these computer systems must be performed manually. Also, the results of those operations must be monitored by looking at files, the display of the system, or other results of performing the operations to determine if the proper result has been accomplished. This is a time-consuming and tedious process. Sometimes this takes the form of releasing a pre-release version of the system, a so-called "beta release." This type of release of a product is one which is distributed to a limited number of individuals, typically very experienced computer program developers, and/or system programmers, to determine if there are any remaining errors or other unexpected results in the underlying system or system software.
Both the initial testing process and the beta testing process, discussed above, are time-consuming and tedious tasks for individuals testing the computer system. They are also expensive for a manufacturer. Such testing requires that the system be tested on a wide variety of compatible platforms (for instance, in the Macintosh brand family of computers, the Macintosh SE brand computer system, the Macintosh II, and other systems using a variety of hardware platforms but having compatible operating system software). For instance, an error which might occur in one model of machine or in one type of software combination may not occur on another machine having a different hardware and/or software combination. Testing each hardware platform in combination with certain underlying software requires careful testing for proper compatibility.
In addition to a variety of manual tests required for ensuring the functioning of a computer system, because the tests are performed manually they may become very time-consuming and tedious. The testing process may also become error-prone, depending on the individuals performing the testing of systems. To ensure that a computer software product is fully functional before shipping a released product, the so-called "beta release" process takes a certain period of time. A long delay in testing a final version of a computer system, including specific hardware and software combinations, may delay the release of a product so much that competitive advantage may be lost.
Some prior art systems used for testing also require that the user input a series of commands and/or user activity which will generate an expected result. These tests must be performed flawlessly in order for the testing process to be performed effectively. In addition, the user must view the tested system's display and/or other results to determine if the user action achieved the desired result. Again, to ensure proper functioning, this must be done repetitively. Human beings typically cannot perform repetitive tasks such as these in an effective manner and monitor all the actions on the tested computer system perfectly. Also, testing of new computer systems requires repetitive and repeatable testing. This is to ensure that errors ("bugs") or other unexpected results are reproducible so that the source of the error can be determined. Therefore, repeatable testing on various hardware/software platforms is required. These tests should also have a minimum impact on the system being tested so as to minimize the effect of the testing process (called "corruption") on the computer being tested. These tests should also have a means for recording the results of the tests so that they may be examined to analyze the error.
In summary, substantial delays and errors are caused in the testing of computer systems prior to shipping. Also, the testing process may fail to catch all errors in a particular hardware/software combination.